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Introduction 


The IDEA Data Center (IDC) Evaluation Work Group developed this document as a guide for IDC technical 
assistance (TA) providers to use when assisting state staff and other stakeholders in planning State Systemic 
Improvement Plan (SSIP) evaluation. We did not develop this document or the corresponding worksheets with 
the intent of providing standalone guidance to states. Instead, we view them as tools for TA providers to use in 
partnership with state staff. We recognize, however, that some state staff and non-IDC TA providers may find 
them useful as well, which is why we are pleased to make them available for download. For direct, individualized 
TA, contact your IDC State Liaison or Tamara Nimkoff (TamaraNimkoff@westat.com) who will connect you with 
an Evaluation TA Specialist. 


The OSEP Guidance and Review Tools for Part B' indicate that a state’s SSIP Phase II evaluation plan should 
specify 


e How the evaluation plan aligns to the theory of action and other components of the SSIP (e.g., data analysis, 
infrastructure analysis) [see Component 3(a)]; 


e How the evaluation includes stakeholders and how information from the evaluation will be disseminated to 
stakeholders [See Component 3(b)]; 


e The methods that will be used to collect and analyze data on the implementation and outcomes of the SSIP 
and the progress toward achieving intended improvements in the SiMR [See Component 3(c)]; and 


e How the evaluation data will be used to examine the effectiveness of the implementation and progress 
toward achieving intended improvements and to make modifications to the SSIP as necessary [See 
Component 3(d)]. 


Below are nine steps with considerations for how to incorporate each of the elements into a well-thought-out 
plan for evaluating an SSIP. 


= 


Understand Phase II evaluation plan in relation to Phase | 

Identify the evaluation team 

Link activities to outputs and outcomes (e.g., by creating an evaluation logic model) 
Develop evaluation questions 

Select an evaluation design 

Identify data collection strategies 

Develop plans for data analysis 


Plan to share and use evaluation results along the way 


SOP OO) OY. es te. 


Develop a timeline 


' U.S. Department of Education, Office of Special Education Programs. (2015, May). Part B State Systemic Improvement Plan 
(SSIP) Phase Il OSEP Guidance and Review Tool. Washington, DC: Author. Retrieved 1/31/202 from 
https://osep.grads360.org/#communities/pdc/documents/8823. 
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Steps in Planning an SSIP Evaluation 


Step 1. Understand Phase II Evaluation Plan in Relation to Phase | 


e Consider how evaluation planning developed in Phase Il aligns with, and in many ways, is an extension of 
the work conducted during Phase 1. The SSIP evaluation will 


° 


Build off of the data analysis. 

= Are useful data available? 

Be informed by the infrastructure analysis. 

= What infrastructure is in place—strengths and challenges? 
Expand and test the theory of action. 

= Is the program logic sound? 

Inform the coherent improvement strategies. 


= What specific actions must the state take to help local education agency (LEA) staff implement 
effective practice? 


e Think broadly about what you want to accomplish with your evaluation. To begin developing or further 
specifying your evaluation plan, start by thinking broadly about what your evaluation may involve. 


° 


° 


° 


° 


° 


° 


What questions do you want and need to answer? 


What data do you need to evaluate implementation and outcomes of the SSIP and the progress toward 
achieving intended improvements in the SiMR? 


Make plans for asking stakeholders what they may want to learn from the evaluation. 
Consider your reporting requirements and the associated timelines. 
What resources does the state have to devote to the evaluation? 


What TA support is needed to plan or implement the evaluation? 


e Once you begin to form a general framework of your evaluation’s purposes and goals, you can start 
planning the evaluation activities in a more detailed manner. 


Step 2. Identify the Evaluation Team 


e Identify the resources you have to conduct the evaluation. 


° 


After previewing what you seek to accomplish with your evaluation, 
= Consider whether you have the expertise in place to plan and conduct the evaluation. 


= Consider what evaluation needs your state may have and how you will seek and access evaluation 
technical assistance as needed. 


= Think about whether you will be working with an internal evaluator, external evaluator, ora 
combination. 


= — If different staff are filling various roles, including an external evaluator, consider how the work will 
be parceled out among multiple evaluator roles. 


= Include individual stakeholders and organizations that will be affected by the evaluation in your 
evaluation planning and possibly on the evaluation team. 


Identify the major evaluation activities you will be conducting (e.g., engaging stakeholders, framing the 
questions, data collection, data analysis) and the expertise and materials that will be required. 


= What resources (e.g., staff, monies, OSEP TA Centers) are available for conducting the evaluation? 
= Are there resource constraints that may require you to adjust your evaluation plans? 
= Are there alternatives that will allow you to carry out the evaluation activities as needed? 


= Are there resources you haven't tapped into, or are there ways to further maximize your existing 
resources? 


e Select the team and define roles. 


° 


° 


Select team members to maximize resources, expertise, and diversity in perspectives (e.g., local agency 
staff, stakeholders). 


Seek external assistance as needed (e.g., external evaluator, technical assistance). 
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Step 3. Link Activities to Outputs and Outcomes (e.g., by Creating an 
Evaluation Logic Model) 


As you plan your evaluation activities, keep your evaluation questions, data collection plans, and analysis plans 
closely tied to your theory of action to ensure you're collecting the data you need. An evaluation logic model can 
help with planning your evaluation activities up front and can help guide the evaluation activities during the 
duration of the project. While OSEP is not requiring that states include a logic model in their SSIP Phase II plan, 
an evaluation logic model can be very useful in evaluation planning as it shows important activities that lead to 
outputs that can be counted and outcomes that can be measured. [See SSIP Evaluation Worksheets 1-4] 

e Consider these questions to help you create an evaluation logic model. 


o What are your resources that go into the program (e.g., fiscal and other investments, project staff, 
organizational partners, stakeholders, technology, evidence-based practices related to intervention and 
implementation)? [Inputs] 


o Whatare the broad approaches you have for realizing the theory of action and addressing the goals? 
[Strategies] 


o Whatare the specific actions planned to implement those strategies? [Activities] 
o What are the immediate results of the project activities? [Outputs] 
o = Whatare the direct results of the activities and their outputs? [Short-term Outcomes] 


o What are the changes in the actions or behaviors based on knowledge or skills acquired through 
outputs? [Intermediate Outcomes] 


o Whatare the results you anticipate that fulfill the SSIP’s goals? [Long-term Outcomes] 


o What is the measurable evidence for achieving each outcome? What are the criteria for successful 
implementation of each outcome? [Performance Indicators/Measures] 


e Refer to your evaluation logic model often as a tool for ensuring that the specified data collection and other 
evaluation activities are occurring and staying on track. [See SSIP Evaluation Worksheet 9] 


o Areall sources of data specified in the logic model being collected according to their respective 
timelines? 


e As adjustments to the program or the evaluation are made during the course of the project, revise and 
update the evaluation logic model as needed. 


Step 4. Develop Evaluation Questions 


Review the project goals, activities, and intended outcomes from the logic model. From these, you can begin to 
identify evaluation questions. 


e =Think about what you want to learn from the evaluation. Questions should address both process and 
outcomes. 


o Process—How’s it going? 
= Are we clear about what we are doing? (gaining clarity) 


= Are we successfully accomplishing our activities? Are we doing what we intended to do 
(monitoring progress by measuring outputs and direct outcomes)? 


= Are we doing things the right way? To what extent did we reach milestones in implementation 
(e.g., # of sites, # of implementers trained to criterion, proficiency on fidelity measures) on schedule, 
gauging progress in implementation of activities? 


= Can we improve what we are doing? What can we do to fix what is not working (focusing on 
improvement)? 


o Outcomes—What good did it do? 


= Whatare the results? Are we making a difference? Did we accomplish our goals? (examining 
impact) 


= Can we show that what we did was responsible for the accomplishments? (linking activities to 
outcomes) 


= How have the state and local infrastructure improved as a result of the SSIP implementation? 
(determining systems change) 
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e With consideration of the questions posed above, think through each significant strategy/activity and its 
related outputs and outcomes to develop a list of evaluation questions specific to the planned intervention 
and the ultimate outcome of interest. [See SSIP Evaluation Worksheet 5] 


e Consider how stakeholders can participate in creating the evaluation questions to be asked and in judging 
the acceptability of the strategies to be used and the value of outcomes to be achieved. 


e Look at your list of evaluation questions and fine-tune your questions, perhaps narrowing the list. 


° 


Will the evaluation question 

= Be of interest to primary intended users? 

= Provide information that addresses the intended use for the evaluation results? 
= Contribute information that is not already known? 


= Be feasible to address, in terms of time, money, resources, and the evaluation team’s skills? 


e Develop specific indicators of progress based on the evaluation questions and linked outcomes. [See SSIP 
Evaluation Worksheet 6] 


° 


Indicators are observable measures of the outcome at the child, student, family, educator, school, 
district, or state level. For example, what will it look like if the outcome is achieved? 


The indicators of progress point to what data you will need to collect and analyze. 


= They may begin with words such as number of, percent of, ratio of, proportion of, mean of, etc. 


Step 5. Select an Evaluation Design 


With your evaluation questions in mind, you can proceed to determining the evaluation design or framework 
you'll use to guide your data collection and analysis plans. [See SSIP Evaluation Worksheet 7] 


e Below are some questions to consider in order to determine your evaluation design. 


° 


What are the objectives of the evaluation? 


= Determine the information needed to make decisions about your SSIP and coherent improvement 
strategies, as well as other information you'll need to know and report. 


What comparisons will be made to demonstrate the effectiveness of the coherent improvement 
strategies? For example, 


= Are you interested in change over time among students/systems (e.g., pre/post or longitudinal)? 


= Are you interested in comparing outcomes for groups of students or families to outcomes for 
others? 


What evaluation design will constitute sufficient evidence for what you need to know? For example, 
= Pre/post data versus longitudinal data with more than two time points? 

= Participants/groups serving as their own control, rather than using other individuals/groups? 

= In-depth, detailed description of the improvement strategies’ implementation and outcomes? 


= Comparison of the outcomes of improvement strategies for one or more groups with groups that 
are similar in terms of their demographic and socioeconomic characteristics but may differ in ways 
not measured? 


= Random assignment to compare the outcomes of improvement strategies for one or more groups 
with an equivalent group or groups that did not receive the improvement strategies? 


If using comparison group data, how will you obtain those data? 


= Are there comparable groups that won't be exposed to the improvement strategies? 


ao Are the data you need already routinely collected for this population/comparison 
group, or would data collection plans need to be developed for this group? 


= — If individuals or groups will be exposed to the improvement strategies at different times, will it be 
possible to collect data from a comparison group before they are exposed to the strategy? 


e As you consider your evaluation design, keep in mind the following: 


° 


° 


° 


The resources available for data collection, management, and analysis; 
The availability and feasibility of comparison groups; and 


The availability of data to measure outcomes. 
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Step 6. Identify Data Collection Strategies 


Specify the methods that the state will use to collect data to evaluate implementation and outcomes of the SSIP 
and the progress toward achieving intended improvements in the SiMR. [See SSIP Evaluation Worksheets 8 and 
9] 
e Consider how each type of evaluation question will be approached. 

o Howwill you measure implementation fidelity? 


o What data will need to be collected to assess whether the coherent improvement strategies are 
associated with changes in outcomes? (test your theory of action) 


o Howwill the evaluation measure state infrastructure changes needed to better align current initiatives 
identified in the infrastructure analysis conducted in Phase I? (measure infrastructure development) 


o How will you evaluate the effectiveness of the support/professional development for LEA staff's 
implementation of evidence-based practices (EBPs)? 


e Consider what data sources and evaluation tools could be used. 


o Think about the extent to which information can be collected and analyzed in a low-cost and practical 
manner. 


o Identify existing data sources. 
= Consider use of data that are already routinely collected. 
= Consider whether someone is available to assist you in accessing data you need. 
= Consider program documents such as logs, records, minutes of meetings. 
o = Think about whether there are already developed instruments that fit your needs. 
= If so, consider what steps need to be taken to obtain permission to use the instruments. 
= Think about what adjustments may be needed. 
o Determine what instruments need to be developed. 
= What value might there be in developing your own tools? 
= Is tool development feasible in terms of time, money, and evaluation team skills? 
e Consider which data-gathering strategies would be best received and most productive. 
o Consider the local district or school culture. 


o Think about who will be your sources of information (e.g., educators, parents, district staff, 
administrators) and how you can maximize their buy-in. 


o Will respondents conform to the methods? For example, will they fill out questionnaires carefully, 
engage in interviews or focus groups, allow observations to be done? 


o Will the information appear as credible to decisionmakers (e.g., funders or policymakers)? 
o Whocan administer the data collection tools? Is training required? 
e Determine your schedule for data collection. 
o What collection schedule is needed to answer the evaluation questions? 
o Are data to be collected as pre/post or at regular intervals? 


e Consider whether you will be conducting a sample, such as a sample of the target students with disabilities 
ora sample of districts or schools. 


o If so, how will you ensure that the sample is representative of all of the participants exposed to the 
coherent improvement strategies? 


Step 7. Develop Plans for Data Analysis 


Specify the methods that the state will use to analyze data to evaluate implementation and outcomes of the SSIP 
and the progress toward achieving intended improvements in the SiMR. 
e Think about data analysis strategies. [See SSIP Evaluation Worksheet 10] 


o Consider potential analyses and how you need to take them into account as you plan your data 
collection activities (e.g., planning for pre/post or comparison group designs). 


o Consider analysis expertise and resources (e.g., available software). 
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e Consider whether the timelines for data collection and analysis fit well with your monitoring and 
reporting needs. [See SSIP Evaluation Worksheets 9 and 11] 


o Are you aware of lags or delays that may affect the availability of certain data? 


Step 8. Plan to Share and Use Evaluation Results Along the Way 


When you develop your evaluation plan, think ahead about how you will share the results and how you 
expect/hope the results will be used. In order to maximize your efficiency and learning, also consider how you, 
as a team, will communicate along the way. [See SSIP Evaluation Worksheet 11] 
e Think about what will make the evaluation useful for yourselves and your stakeholders. 
e Inform and involve people with the organizational authority and power to act on data once it is generated. 
e Consider how often the data will be reviewed for progress monitoring and program improvement. 

o ~=What will this review involve? 

o Whowill participate in the review? 


e Consider how stakeholders will continue to be informed and provided opportunities to weigh in on the 
ongoing implementation of the evaluation. 


e Specify how the state will use the evaluation data to examine the effectiveness of implementation, to assess 
the progress toward achieving intended improvements, and to modify the SSIP as necessary. 


e Determine how changes will be made to the implementation and improvement strategies as a result of the 
data reviews. 


e Determine the process that will be used to modify the SSIP, including the SiMR, as necessary. 


Step 9. Develop a Timeline 


Develop a timeline that identifies important dates based on the evaluation questions you have identified, the 
general evaluation design, your plans for data collection and analysis, and plans for sharing and using results of 
the evaluation along the way. 
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Corresponding Worksheets 


Vv 


SSIP Evaluation Worksheet 1: Definitions 


Theory of action 


An illustration of a program that starts with a vision statement and presents, through if-then statements, a series of strategies that will result in the accomplishment of the 
vision. 


Logic model 


A graphical representation that puts specificity to a theory of action by showing components that are systematically connected to accomplish desired results. A Logic Model for 
Evaluation depicts the key activities, countable outputs, and measurable outcomes on which an evaluation will focus. 


Inputs 


Resources that go into the program 
e Fiscal and other investments e Stakeholders 
e Project staff e Technology 
e Organizational partners e Evidence-based practices related to intervention and implementation 
Examples: Combining initiative with SPDG, Using internal department evaluation specialist 


Strategies 


Broad approaches to realizing the theory of action and addressing the goals 


e Infrastructure development e Scale-up and sustainability 
e Improvement in practices e Analysis and reporting 
Example: Implementation of evidence-based literacy practices using the Reading at School and Home program 


Activities 


Outputs 


Specific actions that implement strategies 
e Actionable plans based on the program’s theory of action 
e Flesh out strategies through concrete events and products 
Example: Establish family partnerships to support student literacy 


Immediate results of the project activities 
e Project accomplishments e Description and number of intervention products/events 
e Customer contacts with products and events e Reports of implementation and results 
e Fidelity of implementation of activities 
Examples: 3 trainings were held, 107 educators attended the trainings 
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SSIP Evaluation Worksheet 1: Definitions (continued) 


Short-term outcomes 


Intermediate outcomes 


Direct results of the activities and their outputs 

e What participants learn as a result of outputs 

e What awareness, attitudes, or skills participants develop 
Example: Parents learn how to monitor their child’s reading 


Changes in the actions or behaviors based on knowledge or skills acquired through outputs 


e Changes in adult actions or behaviors based on knowledge or skills acquired 
e Fidelity of the intervention 
e Improved organizational functioning 
e Improved infrastructure and system functioning 
Example: Increase in number of times parents read with their children 


Long-term outcomes 


The results that fulfill the SSIP’s goals 
Broadest program outcomes 
Results that fulfill the project's goals 
Impact on children or youth with disabilities or their families 
Program scale-up and sustainability 
Example: Increase in student literacy levels 


Performance indicators 


Statement that shows that an outcome is being partially or totally achieved 

Specific, observable, and measurable piece of information 

Concrete evidence that the program has had an impact on a desired outcome 

Evidence can be qualitative or quantitative 

Usually begins with words such as number of, percent of, ratio of, proportion of, mean, or similar phrases 
Examples: 95 percent of teachers measure student reading progress twice a week using (name the measure), 90 percent of families adopt at least one in-home approach to 
read to their child 
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SSIP Evaluation Worksheet 2: Logic Model-Chart Format 


SiMR: 
Strategies/activities Outputs 
A. 
1. (1) : 
(2) 
2, ts) 
(3) 
3. 
B. 
4. (4) ° 
(5) 
(6) , 
(7) . 
(8) 
(9) 
C. ‘ 
(10) P 
(11) - 
D. : 
(12) 
(13) * 
(14) ° 
E. é 
(15) ie 
(16) 
(17) * 
(18) ° 
(19) 


Outcomes 


Short-term: 
e 


Intermediate: 
e 


Long-term: 
e 
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SSIP Evaluation Worksheet 3: Logic Model-Graphical Format (No Arrows) 


SiMR 


Inputs Strategies/activities Outputs 


Outcomes 


Short-term 


cL UL 


Intermediate 


= 
_ 


Long-term 
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SSIP Evaluation Worksheet 4: Logic Model-Graphical Format (Illustrating the Use of Arrows) 


SiMR 


Outcomes 


Strategies/activities Outputs Short-term 


Inputs 


Intermediate 


Long-term 
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SSIP Evaluation Worksheet 5: Outcomes in Relation to Evaluation Questions 


Strategies/activities 


Outputs 


Short-term 


Outcomes 


AVE} IUE-V ace) a Me [Ul=t ae) ay 


Al. 


A2. 


A3. 


Intermediate 


A4. 


A5. 


A6. 


Long-term 


A7. 


A8. 


Short-term 


B1. 


B2. 


B3. 


Intermediate 


B4. 


B5. 
B6. 


B7. 
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SSIP Evaluation Worksheet 5: Outcomes in Relation to Evaluation Questions (continued) 


Strategies/activities Outputs 


Long-term 


AVE-VIUE-V ale) a Me [Ul=t ae) ay 


B8. 


B9. 


ls e Short-term 


2 ° ‘ 


C1. 


C2. 


C3. 


C4. 


Intermediate 


Cs. 


C6. 


C7. 


Long-term 


cs. 


C9. 


C10. 
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SSIP Evaluation Worksheet 6: Outcomes by Evaluation Question and Performance Indicator 


Outcome 


(Short-term, intermediate, 
long-term) Evaluation question Performance indicator 
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Performance indicator 


SIPP Evaluation Worksheet 7: Evaluation Questions in Relation to Performance Indicators, Comparisons Planned, and Evaluation Design 
Evaluation 
question 
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SIPP Evaluation Worksheet 8: Evaluation Questions in Relation to Performance Indicators, Evaluation Design, Data Collection Method, and Tools 


Data collection Data collection tools 
Evaluation question Performance indicator Evaluation design method (instrument/protocol) 
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SSIP Evaluation Worksheet 9: Evaluation Data Collection Schedule 


Status of instrument/ 
rece} colece) 


E=exists 
Evaluation UD=under development Frequency of 
question # Instrument/protocol =] DE fold of-Mol-\V/-1 (of of-Lo| (oto) | (=el uel} First data collection Nelelifelarel merle mee) (rad (elary 
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SSIP Evaluation Worksheet 10: Evaluation Analysis Plans by Evaluation Question, Performance Indicator, and Instrument 


Evaluation Data collection tool 

question # Performance indicator (instrument/protocol) Planned analysis 
e e e 
e e e 
e e e 
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SSIP Evaluation Worksheet 11: Plan for Data Use and Dissemination by Analysis Results 


Planned analysis Process for use for program improvement Possible implications for ongoing evaluation Plans for disseminating results to stakeholders 


A2. : 
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